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In a manner similar to the molecular chaos that underlies the stable thermodynamics of gases, 
neuronal system may exhibit microscopic instability in individual neuronal dynamics while a macro¬ 
scopic order of the entire population possibly remains stable. In this study, we analyze the micro¬ 
scopic stability of a network of neurons whose macroscopic activity obeys stable dynamics, express¬ 
ing either monostable, bistable, or periodic state. We reveal that the network exhibits a variety 
of dynamical states for microscopic instability residing in given stable macroscopic dynamics. The 
presence of a variety of dynamical states in such a simple random network implies more abundant 
microscopic fluctuations in real neural networks, which consist of more complex and hierarchically 
structured interactions. 


I. INTRODUCTION II. RANDOM NEURAL NETWORK 


While an animal is repeating a fixed action in response 
to a given stimulus, individual neurons in the brain do 
not necessarily reproduce identical activity [H-Q- The 
contrast between the reliable animal behavior and the 
erratic activity of single neurons may be compared with 
thermodynamics of gases, in which macroscopic states 
obey thermodynamic laws with small degrees of free¬ 
dom, while individual molecules obey chaotic dynamics 
involving large degrees of freedom. In the thermody¬ 
namics, the difference in stability is resolved in such a 
way that macroscopic thermodynamic laws are deduced 
through the Boltzmann equation describing the micro¬ 
scopic chaotic motion of simplistic model molecules 
It has been one of key objectives of statistical physics to 
relate microscopic dymanics with macroscopic dynamics 
in diverse phenomena, including populations of active el¬ 
ements [a-llOll and human activity |lll - [l6l| . 

For networks of randomly connected neurons, it has 
been shown that a macroscopic order parameter repre¬ 
sented by the average neuronal activity obeys determinis¬ 
tic dynamics; in particular, networks of McCulloch-Pitts 
binary neurons [l7| exhibit three distinct types of macro¬ 
scopic dynamics, exp ressing either monostable, bistable, 
or periodic state [1^. By contrast, in the same system, it 
was revealed that a microscopic state specified by a set of 
individual neuronal states may become unstable against 
microscopic perturbations, such as flipping a single neu¬ 
ron state [l9|. However, in that study, the microscopic 
instability was verified solely by numerically simulating 
small systems, and accordingly, the mechanism of the 
instability was not examined thoroughly. 

Here, we study the microscopic instability of neural 
networks in detail using analytical as well as numerical 
analysis, and reveal various types of microscopic dynam¬ 
ics residing in stable macroscopic dynamics. 
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We consider a network of McCulloch-Pitts binary neu¬ 
rons interacting via random synaptic connections. Here, 
we adopt a symmetric expression with active and inactive 
states respectively represented as 

Mt) = I +1; (1) 

where i{= I, 2, • • • , N) is the label of a neuron and t is the 
discretized time given by an integer. In every time step, 
all the states of an entire neuronal population are up¬ 
dated synchronously, such that each neuron is either ac¬ 
tivated or deactivated depending on whether the summed 
input exceeds the threshold or not: 

N \ 

'y ^ u'ijSj(t) -\- hi \, (2) 

/ 

where sgn(a:) is the sign function, N is the total number 
of neurons, Wij represents the synaptic connection from 
the jth neuron to the ith neuron, and —hi is the threshold 
for the fth neuron (Fig.[T]). 

The state evolution rule Eq. is similar to the zero- 
temperature relaxation dynamics for a spin glass given 
by the exchange interactions Wij and external fields 
hi. Characteristics that distinguish our neural network 
model from spin systems are as follows: (i) updates are 
synchronous, (ii) connections Wij are generally asymmet¬ 
ric, and (hi) self-connections wu can be present. We de¬ 
note synaptic connections as Wij to distinguish from the 
exchange interactions of spin systems, usually denoted as 
Jij. 

Note that the neuronal states can alternatively be ex¬ 
pressed as 1 and 0, a straightforward representation of 
the active and inactive states, respectively. This can 
be done by transforming as Ui{t) = {si{t) -\- l)/2. Ac¬ 
cordingly, the state evolution Eq.© may be rewritten 

as Ui{t -I- 1) = , where 9{x) is the 

Heaviside step function. In this case, the threshold Ti is 
given by {-hi -k w^J)/2. 
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with possible fluctuations in 0{-\/N). In the limit of a 
large number of neurons, iV —>■ oo, the activity level in 
the next step is determined as TO(t + 1) = 2Q/N — 1 in 
terms of the current activity level rn(t), thus forming the 
evolution equation, 


, _ ^ / wmit) -\- h\ 

TO(t + l) = erf|^--^=-j, (6) 

where erf (a;) is the error function defined by 

erf(a:) = [ dve~^ . (7) 

Jo 


FIG. 1. A recurrent network of neurons. Wij represents a 
synaptic connection from the jth neuron to the ith neuron. 
Neurons update their states according to Eq. 0). 


III. MACROSCOPIC DYNAMICS 

One of the authors has shown that a macroscopic ac¬ 
tivity of the neural network obeys deterministic dynamics 
in the limit of a large number of neurons, N ^ oo [3. 
Here, we derive the evolution equation of a network 
whose synaptic connections Wij are drawn independently 
from an identical Gaussian distribution with a mean of 
w/N and a variance of 1/N, whereas the threshold is 
chosen as a constant —hi = —h. 


A. Evolution equation of macroscopic activity 

Consider the situation in which a set of states {si(t)} 
is selected randomly under a given mean activity, 

1 ^ 

Mt) = 

If the neuronal states are statistically independent of the 
synaptic connections {wij}^ inputs to individual neurons, 
given as 

N 

Vi = '^WijSj{t) + h, (4) 

f=i 

are expected to distribute normally with a mean of 
wm(t) + h and a variance of 1. Thus, the total num¬ 
ber of neurons that will be activated in the next step will 


Because neuronal states in the next step {si(t-|-1)} are 
determined by the set of synaptic connections {wij}, they 
are generally not independent of the connections. Nev¬ 
ertheless, the network activity keeps following the evo¬ 
lution of Eq.® if the network is of a reasonably large 
size [3 - The evolution equation may show three types 
of dynamics depending on macroscopic parameters w and 
h: monostable (Sm), bistable (Sb), or periodic (P) states 

(Fig. [21). 



w 

FIG. 2. Phase diagram of the macrodynamics in a plane of w 
and h. P: periodic state, Sm: monostable state, Sb: bistable 
state. 
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(a) 



(b) 



(c) 



B. Stability of macroscopic dynamics 


The evolution equation Eq.([6]) may have a fixed point 
m{t) = m that satisfies the self-consistent equation, 


m = erf(i). (8) 

where / is the average input given as, 

/ = wm h. (9) 


The macroscopic activity is stable if the absolute slope 
of the iteration map Eq.® at the point of intersection 
with the y = X line is smaller than unity: 


w 



< 1 . 


( 10 ) 


The system is called monostable if the evolution equation 
has only one stable fixed point (Eig. |3Da)). 

By increasing the mean synaptic connection w from the 
monostable regime, the fixed point loses stability when 
the slope of the iteration map Eq.® at the intersec¬ 
tion becomes greater than 1. The system then becomes 
bistable, thorough a pitchfork bifurcation, which involves 
a pair of stable fixed points appearing on both sides of the 
destabilized fixed point (Fig. EKb)). The boundary be¬ 
tween the monostable and bistable regimes is obtained by 
solving w = y/ir12 exp (/^/2) with m satisfying Eq.®. 

Contrariwise, by decreasing w from the monostable 
regime, the single fixed point loses stability when the 
slope of the iteration map Eq.® at the intersection be¬ 
comes smaller than —1. The system then begins to oscil¬ 
late through a period-doubling bifurcation (Fig.[3Kc)). In 
this periodic state, the macroscopic activity m{t) oscil¬ 
lates between the two newly appeared stable fixed points 
of the iterated map, 


m = erf 


w erf 


( wm-\-h ^ 

C2 J 


V2 


( 11 ) 


FIG. 3. Examples of the three types of dynamics of the macro- IV. MICROSCOPIC DYNAMICS 

scopic activity: (a) monostable {w = 1, h = 0.5), (b) bistable 

(re — 2, h — 0), and (c) periodic state (w ~ 2, h — 0). While the macroscopic order parameter m(t) exhibits 

stable dynamics following the simple iteration map 
Eq-®, it is possible that a set of neuronal states 
are dynamically changing in time within the given con¬ 
straint, m{t) = (1/V) Si(t)±0(l/-\/]V). One of the 

authors has numerically examined the possibility that the 
system may be microscopically unstable due to the state 
flipping of one neuron [l9j . In the present study, we ana¬ 
lytically estimate the parameter range of the microscopic 
instability. 























4 


A. Microscopic instability in the macroscopically 
stable regimes 


We first examine the microscopic stability of the 
macroscopically stable regime, including the monostable 
and bistable states. While microscopic states evolve with 
the individual neuronal dynamics of Eq. ([2|) , we consider 
flipping a single neuron state and examine whether the 
flipping spreads over the network or not. By flipping the 
state of the pth. neuron, input to the ith neuron is altered 
from Vi = + h to 


N 

+ h, 


( 12 ) 


where Sjp is the Kronecker delta. The state of the zth 
neuron will be altered in the next step if the sign of the 
input is reversed: 


^ 

Vi/P / 


2 

- w'^p < 0 . 


(13) 


Under the assumption that {wij} and {s^} are indepen¬ 
dent, the probability P at which the above mentioned 
inequality holds is obtained analytically. Because the 
first and the second terms in the RHS of Eg. dTSl) are 
distributed normally, the probability P is given as 


P = 


N 


2ttVN - 1 



dxdy exp — 


N{x-§y 


N{y-f + ^y 

2 (TV - 1) 


(14) 


In the limit of a large number of neurons, P is approxi¬ 
mated as 





Nx'^ \ 

2 J 

'-\x\ - f\ 

. V 2 J 



(15) 

(16) 


The system is microscopically unstable if the flipping 
spreads from a single neuron to more than one neurons, 
i.e., if NP > 1. This implies that the microscopic state 
remains unstable under a given stable macroscopic or¬ 
der. This instability condition is summarized in terms of 
average input / = wm -|- h as 


I/I <Ic = 




(17) 


With this condition, the macroscopically monostable 
regime can further be classified into two regimes on the 



FIG. 4. Microscopic stability of the macroscopically stable 
states {N = 1000). A: Both macroscopically stable states are 
microscopically stable. B and B’: One of two macroscopically 
stable states is microscopically unstable. C: Both macroscop¬ 
ically stable states are microscopically nnstable. D and D’: 
The macroscopically monostable state is microscopically sta¬ 
ble. E: The macroscopically monostable state is microscopi¬ 
cally unstable. 


basis of whether the system is microscopically unstable or 
stable. In the bistable regime, in which the system may 
perform an alternative mean activity, the microscopic in¬ 
stability of the system depends on the macroscopic state. 
Thus, the bistable parameter regime can be further clas¬ 
sified into four regimes on the basis of whether individual 
macroscopic states are microscopically unstable or stable. 
The categorized areas are depicted in Fig. SI 


B. Microscopic instability in the macroscopically 
periodic regime 


Next, we examine the microscopic instability of the 
system whose macroscopic activity is oscillating with pe¬ 
riod two. In the first half of the period two, m is mapped 
to m' = erf (//y^) through mean inputs / = wm h. 
In the second half, m' returns to m through mean inputs 
g = wm' + h as m = erf (g/ y^). Accordingly, g and / 
are mutually bounded as 

g = w evt{f / \/2) + h, (18) 

f = weTi{g/\/2)h. (19) 
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It follows from Eas. dTSl) and (IT9t that 

f- 9 


h = 


eii{g/y/2) -eri{f/y/2)’ 
feri{f/V2) - gerf{g/V2) 
eri{f/V2) - er^{g/^/2) 


( 20 ) 

( 21 ) 



FIG. 5. Microscopic stability of macroscopically periodic 
states {N — 1000). F: Both macroscopic states are micro¬ 
scopically unstable. G and G’: One of two macroscopic states 
is microscopically unstable. H: Both macroscopic states are 
microscopically stable. 


The system exhibits microscopic instability if both av¬ 
erage inputs I/I and \g\ are smaller than the critical 

value Ic = <^2\og The microscopically unsta¬ 

ble regime of parameters (u), h) is obtained by search¬ 
ing them under the constraints |/| < Ic and < Ic in 
Eas. d^ and (1^ . In addition to this perfect instabil¬ 
ity regime, there are regimes in which either of the two 
states m and m' is unstable, such that |/| < Ic < |g| or 
I 5 I < Ic < |/|. The categorized regimes are depicted in 
Fig. [5] 

Notably, the microscopic instability defined by the sta¬ 
bility against a one-neuron flip is dependent on the num¬ 
ber of neurons N, as in Eq. (HU, and the microscopic 
instability region may expand without bound. How¬ 
ever, the dependence follows the square root of a log¬ 
arithm, v^log N, and accordingly, the instability range 
stays in a small range even in a large network consisting 
of 0(10^) - 0(10®) neurons (Fig. i]). 



w 

FIG. 6. Boundaries of microscopic stability. The boundaries 
for ^=1000, 10000, and 100000 are depicted in red, brown, 
and purple, respectively. Regions A to H correspond to the 
respective regions given in Figs. 0] and 


V. NUMERICAL SIMULATION 

A variety of microscopically unstable phases revealed 
by the current analytical consideration were not observed 
by the previous simulations of small networks, which 
ranged from A = 20 to 200 [l^. The advancement of 
computers in recent decades has enabled us to simulate 
the larger networks. Here, we show the results of simu¬ 
lating a network of size N = 1000. 

Using a simplified model, we simulated the evolution 
Eq.® of neurons interacting through synaptic connec¬ 
tions distributed normally. Given an initial condition, we 
iteratively applied the evolution equation for 5000 steps, 
expecting that the system would attain macroscopically 
stable activity. Then the system, starting with the final 
state, was iterated for one more step. In addition, we 
flipped a single neuron from the final state and iterated 
the system for one step. We decide whether the system 
was microscopically stable or unstable, on the bases of 
whether the neuronal states of these two systems were 
entirely identical or not. 

Figure [7] represents the parameters that make the sys¬ 
tem unstable. In the macroscopically monostable regime, 
the numerically verified range of microscopic instability 
is consistent with the theoretical range. In the bistable 
regime, microscopic stability depends on an alternative 
macroscopic state. The macroscopic state may be suit¬ 
ably selected by choosing a proper initial condition. For 
instance, a macroscopic state of higher activation would 
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likely be selected if we choose the initial condition with 
all the neurons activated, = 1}. Figure [5] depicts 
the microscopically unstable regime obtained from this 
all-active initial condition, which is consistent with the 
analytical result. 



w 


FIG. 7. Microscopic instability obtained by numerical sim¬ 
ulation of the networks of = 1000. A set of parameters 
(w, h) with which the network exhibit instability is depicted 
as a dot. Initial states of neurons were chosen at random, 
Si = ±1. 

In the periodic regime, microscopic instability may de¬ 
pend on the timing of flipping occurrences. The system 
remains stable if a neuron is flipped in the stable phase 
of the oscillation. The microscopic states may deviate 
if a neuron is flipped in the unstable phase, but these 
systems may merge in the next step. The numerically 
obtained unstable regime is similar to the range in which 
both states m and m' are unstable. 


VI. EVOLUTION OF THE MICROSCOPIC 
DISTANCE BETWEEN TWO STATES 



FIG. 8 . Microscopic instability obtained from the set of initial 
states of {si = 4-1} {N = 1000). 


which possess the identical macroscopic activities m = 
expressing the identical macroscopic dynam¬ 
ics, obeying Eq. Because of the huge amount of com¬ 
binations of the individual neuronal states, the two states 
can possibly remain microscopically unidentical. We es¬ 
timate the evolution of the microscopic distance of two 
states measured with the normalized Hamming distance, 

p4) 


Here, we consider the case in which the macroscopic 
activity m = is stable. We assemble all possi¬ 

ble microscopic states s'^(t) and s^(t) with macroscopic 
activity is m and that the mutual distance is d, and es¬ 
timate the distribution of the distance in the next time 
step. 


Dynamical aspects of microscopic states may also be 
captured by analyzing the evolution of the distance be¬ 
tween two microscopic states. The map of the distance 
has been obtained for a specific case of w = 0 and 
h = 0 (20l - [^ . Here we extend the analysis to general 
cases of arbitrary w and h. 

Consider the situation that the two states 


1 5 

.. 

*2 5 

■ I ■Sjv}; 

( 22 ) 

.B 
1 1 

•• 

•,4}, 

(23) 



(25) 

N 

d' = ^ \sf{t -1- 1 ) — sf (t -b 1 ) , 

7=1 

(26) 

. N 

- 1 ^ 

II 

(27) 


The mean distance in the next step is obtained as a func- 
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tion of the distance in the current step. 


u 


V 

ip{d) 


N 

Wij sf + h a = A, B , 


uf + uf 
2 v^(r^’ 
ut - uf 
2Vd ’ 

(d') = Prob[ufuf < 0] 


Prob 



Id.. 

u < \ 

- -: b 

V 

1 - d ' ' 


(28) 

(29) 

(30) 

(31) 

(32) 


Under the assumption that Wij and sf are independent 
and Wij are normally distributed, the RHS of Ea. (|32l) is 
obtained in an integral formula as 


ip{d) = 


1 

27r 



dudv X 


exp 



2'/^ J -c 


dv exp ( —- 


',rf ( 


— erf 


-Vd\v\ - f \ \ 

) i 


(33) 

(34) 

(35) 

(36) 


Note that the microscopic distance is bounded as d < 
dmax = 1 — \m\ for m = . 

Figure [S] represents the evolution map of the distance 
(/ 3 (d) for several values of h with w kept ar a value of 0 . 
The map of the case u; = d = 0 is (/ 3 (d) = ^ sin“^ \fd^ as 
has been obtained in Ref. [13, mi. 

When '/d « f, 



dv Vd I 


v\ exp 





(37) 

(38) 


The microscopic distance of d = 1/7V corresponds to flip¬ 
ping a single neuron. In this case, the average distance 
in the next step (p{l/N) represents the probability of any 
other single neuron flipping due to the first single neuron 
flipping. Thus, the condition for microscopic instability 
discussed in the last section, NP > 1 , is identical to the 
condition of 






(39) 


Figure [10] represents the manner in which the mapping 
ip{d) varies as the parameters cross the microscopic in¬ 
stability line. It was found from the distance mapping 
(/3(d) that the instability of this system is not the linear 
instability in which the gradient of the mapping exceeds 
unity, but is simply caused by the inequality Fa. (|5^ . 



FIG. 9. Distance maps, (/ 3 (d) for various h (w = 0). 

VII. THE PERIOD OF THE MICROSCOPIC 
STATE ATTRACTOR 


Because the total number of microscopic states is of 
finite 2^, and the dynamics of individual neurons, de¬ 
scribed in Eq.([2]), are deterministic, the system eventu¬ 
ally enters a cyclic orbit. It has been numerically de¬ 
termined that the period of the attractor cycle of the 
random neural network of (tcUi) = ( 0 , 0 ) increases expo¬ 
nentially with N, on average [l9|. By simulating systems 
of a size larger than that of the previous study, we con¬ 
firmed that the logarithm of the periods fits to a linear 
function of N fairly well (Fig. [TT|) . 

(log T) Ri yiV-I-c, (40) 

where log is the natural logarithm. The linear regression 
analysis applied to the simulation data of V = 15 to 31 
gives the coefficient 7 ~ 0.216 ± 0.002. Note that this 
period is significantly shorter than the average period 
of the random Boolean map, also called the Kauffman 
map, which is obtained analytically as T ~ and the 
Boolean map’s exponent is jb = (log2)/2 Ri 0.347 [^ . 
Thus, the period of the microdynamics of the random 
neural network of {w, h) = ( 0 , 0 ) typically grows expo¬ 
nentially with iV, but is shorter than that of the random 
Boolean map. 

We now examine how the exponent 7 changes with the 
model parameters. Figure [T^l represents the manner in 
which the exponent 7 changes with h, while w is kept at 
0. In the microscopically stable regime, \h\ > Ic, where 
(p{l/N) < 1/N or the fixed point of the distance map 
satisfying (p{d*) = d* is less than 1/N, the period of the 

























(a) 




w 

FIG. 10. Change in the distance map </ 5 (d) across the micro¬ 
scopic stability line {N — 1000). (a) Three kinds of distance 
maps, (b) Model parameters for the three distance maps. 


microscopic attractor T is expected to be unity, implying 
the average logT is close to zero. 

Contrariwise, in the microscopically unstable regime, 
\h\ < If., where the fixed point in the distance map d* 
is greater than 1/iV, the microscopic state is expected 
to meander in state space, and the period of the micro¬ 
scopic attractor T is exponentially large. The number of 
microscopic states given in the range of the distance d* 



FIG. 11. Period of microscopic states and the number of 
neurons. 



FIG. 12. Dependence of the exponent 7 on the model param¬ 
eter h (w = G, N = 30). 7 s is the exponent estimated by the 
Boolean random map, given a typical number of microscopic 
states. 

is roughly estimated as 



where [a;J represents the floor of x. When considering 
the random Boolean map among W states, the typical 
length of the attractor period is vW, and accordingly 
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the exponent 7 _b is given as 

7 b = ^ log W. (42) 

Figure [12] compares the jb and the real exponent 7 esti¬ 
mated by numerical simulation. Though the exponent jb 
on the basis of the random Boolean map overestimates 
the actual exponent 7 , the dependence on the model pa¬ 
rameter h is qualitatively reproduced. 


VIII. DISCUSSION 

We have analytically and numerically examined the 
microscopic dynamics of randomly connected neural net¬ 
works, and revealed a variety of microscopic dynamics. 
A network that exhibits stable dynamics in its macro¬ 
scopic activity may show instability in its microscopic 
state, as is suggested by the real neural irregular activity 
in a fixed behavioral context. The analysis of a simplistic 


system could provide a possible link to the real system. 
In other words, the real neural network expressing the 
nonreproducible activity of individual neurons in a fixed 
behavioral response may represent the microscopic insta¬ 
bility in the macroscopic stable dynamics. 

It should be noted that a neural network expresses mi¬ 
croscopic instability in the entire parameter region in the 
limit of a large number of neurons, which corresponds to 
the thermodynamic limit of gases. Thus the coexistence 
of microscopic instability with macroscopic stability is 
expected to play an important role in the information 
processing in the real neuronal circuitry consisting of a 
huge number of neurons. 
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